Method and system for low-field mri denoising with a deep complex-valued convolutional neural network

ABSTRACT

Blurring and noise artifacts in magnetic resonance (MR) images caused by off-resonant image components may be corrected with convolutional neural networks, particularly feed forward networks with skip connections. Demodulating complex blurred images with off-resonant artifacts at a selected number of frequencies forms a respective real component frame of the MR data and a respective imaginary component frame for each image. A convolutional neural network is used to de-blur the images. The network has a plurality of residual blocks with multiple convolution calculations paired with respective skip connections. The method outputs, from the convolutional neural network, a de-blurred real image frame and a de-blurred imaginary image frame of the MR data for each complex blurred image.

CROSS-REFERENCE TO RELATED APPLICATION

This Application claims priority to and benefit of U.S. ProvisionalPatent Application Ser. No. 63/333,232 entitled “Method and System forAutomatic Off-Resonance Correction for Spiral Imaging with aConvolutional Neural Network,” filed Apr. 21, 2022, which is herebyincorporated by reference herein in its entirety as if fully set forthbelow.

STATEMENT OF GOVERNMENT INTEREST

This invention was made with government support under Grant No.EB028773, awarded by the National Institutes for Health. The governmenthas certain rights in the invention.

FIELD

The present disclosure relates to systems and methods for denoisingmagnetic resonance images using complex de-noising convolutional neuralnetworks.

BACKGROUND

Magnetic resonance imaging (MRI) is an important diagnostic tool forvarious conditions, including brain conditions. Because of the good softtissue contrast, non-invasiveness and lack of ionizing radiation of MRI,it is widely used as a diagnostic tool for brain conditions includingstroke, tumors, multiple sclerosis (MS), hemorrhage, blood vesselissues, and neurodegenerative diseases. A clinical protocol oftenincludes pre- and post-contrast T1, T2, fluid-attenuated inversionrecovery (FLAIR), proton density (PD) and diffusion weighted images(DWI). Other advanced sequences such as magnetic resonance angiography(MRA) and perfusion MRI using dynamic susceptibility contrast (DSC) andarterial spin labelling (ASL) are also used for specific conditions.Although different contrasts provide enriched diagnostic information,the challenges are prolonged scan time and increased artifacts due tomotion [1], especially for pediatric patients who have trouble holdingstill during the scan and thus sedation/anesthesia is often needed for asuccessful exam [2].

Extensive studies have been performed to accelerate MRI and reducemotion artifacts with the application to the brain and other regions ofthe body. Most studies have focused on improving the data acquisitionstrategies, such as using partial Fourier [3], parallel imaging [4-6]and compressed sensing [7] to reduce the number of acquired k-spacelines without introducing aliasing, many of which are already widelyused in clinical protocols. Retrospective [8] and prospective [9] motioncorrection methods have also been developed to specifically reduce theartifacts. However, these acquisition strategies often introducetrade-offs among speed, resolution and image quality, which is typicallyevaluated by the real or apparent signal-to-noise ratio (SNR), so thatfurther acceleration of the scan can lead to reduced SNR and/or spatialresolution. In clinical practice, these aspects are balanced to yield astandard protocol. Denoising algorithms, which are applied duringpost-processing, can increase SNR without introducing any negativeeffects to the acquisition process and thus have the potential to shiftthe balance towards more aggressive acceleration and compensate for thereduced SNR in the original images.

Denoising algorithms can improve signal-to-noise ratio (SNR) withoutprolonging the scan time. Filter-based denoising methods, such asnon-local means (NLM) and block-matching and 3D filtering (BM3D), sufferwhen dealing with small lesion regions and non-uniform noise patternsdue to parallel imaging and B1 inhomogeneity from multiple coils.Recently deep convolutional neural networks have been developed fordenoising; however, they require high-quality training data, which isdifficult to obtain in practice. The networks are usually trained withthe noise-corrupted images as the input and the noise-reduced/noise-freeimages as the output. The input can be simulated from the output imagesby adding noise at one or multiple levels with the desired distributionor from actual images acquired with low SNR. The DCNN can then learnfrom the “examples” to achieve good denoising when the new images aresimilar to those in the training dataset. In addition to its improvedperformance, the DCNN is also much faster to run as only one forwardpass is required once it is trained.

In clinical practice, a clear MRI with high signal to noise ratio (SNR)is usually favored for accurate lesion detection and diagnosis.Improving the SNR of MRI can be achieved by changing the parameters ofacquisition sequences such as using more averages and lower bandwidth;however, this often comes with prolonged scan time. On the contrary,improving SNR with denoising algorithms during post-processing would notchange the scan process and therefore is an attractive alternativeoption. Most of the denoising algorithms can be categorized astraditional filter-based methods and learning-based methods.Filter-based methods, including non-local means (NLM) and block-matchingand 3D filtering (BM3D) [11], often rely on repetitive structures in theimages so that local or global averages can be applied to reduce noise.The main disadvantages of these methods include the following: 1) alarge number of similar structures need to exist in the input images toachieve good performance, which can become problematic for finestructures and pathological regions as fewer such blocks exist; and 2)the performance is highly dependent on algorithm parameters, which canvary significantly for different sequences and noise levels, especiallywhen advanced image acquisition methods, such as parallel imaging withmultiple receiver coils, are used, as the noise distribution is muchmore complicated.

Now with reference to prior art FIG. 3 , a U-Net is a convolutionalneural network architecture. U-Nets may be effective for tasks where theoutput is of similar size as the input and the output needs a similarlevel of spatial resolution. This makes a U-Net effective for superresolution image processing. To perform classification using aconvolutional neural network the image is down-sampled into one or moreclassifications using a series of stride two convolutions reducing thegrid size each time. To be able to output a generated image of the samesize as the input, or larger, an up-sampling path is used to increasethe grid size.

The up-sampling/decoder path may include several transposed convolutionscan be used, where each transposed convolution adds pixels between andaround the existing pixels. Each up-sample in the decoder/up-samplingpart of the network can add pixels around the existing pixels and alsobetween the existing pixels to eventually reach the desired resolution.Replication padding is then performed to provide an extra pixel aroundthe image. Then average pooling can be performed to extract featuressmoothly. After new pixels are added, the subsequent convolutions canimprove the level of detail as the path continues through the decoderpath of the network an upscaling step increases the dimensions of theimage.

The 3D UNet was originally proposed by Cicek et al. for automaticsegmentation of Xenopus (a highly aquatic frog) kidney. It has anencoder-decoder style architecture with skip connections betweencorresponding layers in encoding and decoding paths. This architectureis very popular for medical image segmentation. FIG. 3 shows the blockrepresentation of 3D UNet architecture.

Each convolutional block has two convolutions followed by max pooling.Every convolution is immediately followed by a rectified linear unit(ReLU) activation and batch normalization layer. Each deconvolutionalblock consists of two convolutions followed by a deconvolution to regainspatial dimension. Moreover, there are skip connections from theencoding path to decoding path at corresponding spatial dimensions.These are shown by green arrows. The very final convolution generates athree-dimensional feature map and is followed by activation in order toobtain a pseudo-random probability distribution at each pixelrepresenting its class membership.

Deep convolutional neural networks (DCNN) with various architectureshave

yielded performance superior to traditional methods [13]. These networksare usually trained with the noise-corrupted images as the input and thenoise-free images as the target output. The DCNN can then learn from the“examples” to achieve good denoising when the new images are similar tothose in the training data. However, a disadvantage is the sole relianceon the training data, or good “examples”, which are difficult to obtainin practice. Simulating low SNR images by adding noise often uses a verysimplified noise model with a spatially uniform Gaussian or Riciandistribution, and thus cannot represent more complicated cases withnon-uniform noise from multiple coils. Acquiring paired low and high SNRimages can overcome this issue but suffers from any mismatches betweenthe two acquisitions. As the number of training examples from eachsequence type may need to be large and diverse to obtain goodperformance, the data collection can be challenging and expensive.Furthermore, if a sequence type is not in the training set, it isdoubtful whether the model can generalize to this sequence. In order tosolve the problem of over-dependence on training data, an unsuperviseddeep convolutional neural network (U-DCNN) that does not requiretraining from “examples” but relies on different characteristics of thenetwork against signal and noise was recently proposed and producedcompelling results on denoising natural images [14].

BRIEF SUMMARY OF THE DISCLOSURE

This disclosure includes, in one implementation, a computer-implementedmethod of denoising a magnetic resonance (MR) image. The first step ofthe method includes acquiring complex magnetic resonance (MR) image dataof an area of interest of a subject, wherein the image data includescomplex blurred images of multi-coil MR image data, and wherein thecomplex blurred images include resonant image data and off-resonanceartifact data. For each of the complex blurred images, the methodincludes demodulating the complex blurred images at a selected number(n) of frequencies to form, for each of the n frequencies, a respectivereal component frame of the MR data and a respective imaginary componentframe of the MR data. Compiling a layered data set includes stacking therespective real component frames and the respective imaginary componentframes. The method uses the layered data set as an input to aconvolutional neural network (CNN) having a plurality of residualblocks, wherein the residual blocks incorporate multiple convolutioncalculations paired with respective skip connections. The methodoutputs, from the CNN, a de-blurred real image frame and a de-blurredimaginary image frame of the MR data for each complex blurred image.

In another embodiment, a system for denoising a magnetic resonance (MR)image, includes one or more processors and a memory device coupled tothe one or more processors and storing instructions which, when executedby the one or more processors, cause the system to perform functionsthat include a computer-implemented method of denoising a magneticresonance (MR) image. The first step of the method includes acquiringcomplex magnetic resonance (MR) image data of an area of interest of asubject, wherein the image data includes complex blurred images ofmulti-coil MR image data, and wherein the complex blurred images includeresonant image data and off-resonance artifact data. For each of thecomplex blurred images, the method includes demodulating the complexblurred images at a selected number (n) of frequencies to form, for eachof the n frequencies, a respective real component frame of the MR dataand a respective imaginary component frame of the MR data. Compiling alayered data set includes stacking the respective real component framesand the respective imaginary component frames. The method uses thelayered data set as an input to a convolutional neural network (CNN)having a plurality of residual blocks, wherein the residual blocksincorporate multiple convolution calculations paired with respectiveskip connections. The method outputs, from the CNN, a de-blurred realimage frame and a de-blurred imaginary image frame of the MR data foreach complex blurred image.

In yet another embodiment, a non-transitory computer-readable mediumstores instructions thereon which, when executed by one or moreprocessors, cause a computer to perform functions for denoising amagnetic resonance (MR) image that include acquiring complex magneticresonance (MR) image data of an area of interest of a subject, whereinthe image data comprises complex blurred images of multi-coil MR imagedata, and wherein the complex blurred images comprise resonant imagedata and off-resonance artifact data. The instructions take each of thecomplex blurred images and cause the computer to perform functions thatinclude a computer-implemented method of denoising a magnetic resonance(MR) image. The first step of the method includes acquiring complexmagnetic resonance (MR) image data of an area of interest of a subject,wherein the image data includes complex blurred images of multi-coil MRimage data, and wherein the complex blurred images include resonantimage data and off-resonance artifact data. For each of the complexblurred images, the method includes demodulating the complex blurredimages at a selected number (n) of frequencies to form, for each of then frequencies, a respective real component frame of the MR data and arespective imaginary component frame of the MR data. Compiling a layereddata set includes stacking the respective real component frames and therespective imaginary component frames. The method uses the layered dataset as an input to a convolutional neural network (CNN) having aplurality of residual blocks, wherein the residual blocks incorporatemultiple convolution calculations paired with respective skipconnections. The method outputs, from the CNN, a de-blurred real imageframe and a de-blurred imaginary image frame of the MR data for eachcomplex blurred image.

Other aspects and features according to the example embodiments of thepresent disclosure will become apparent to those of ordinary skill inthe art, upon reviewing the following detailed description inconjunction with the accompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

Reference will now be made to the accompanying drawings, which are notnecessarily drawn to scale.

FIG. 1 is a system diagram illustrating an operating environment capableof implementing aspects of the present disclosure.

FIG. 2 is a computer architecture diagram showing a general computingsystem capable of implementing aspects of the present disclosure.

FIG. 3A is a PRIOR ART schematic that illustrates a conventional U-Net.

FIG. 3B is a PRIOR ART flow chart of a method of using a Non-UniformFast Fourier Transform (NUFFT) on MRI images in accordance withembodiments of this disclosure.

FIG. 4 is a PRIOR ART schematic that illustrates a de-noisingconvolutional neural network DnCNN.

FIG. 5 is a schematic representation of a computerized method ofproducing blurred MR images to train a neural network according toimplementations of this disclosure.

FIG. 6 is a schematic representation of utilizing a convolutional neuralnetwork on frames of real and imaginary MR image data to de-blur an MRimage according to aspects of this disclosure.

FIG. 7A is a frame of MR image data taken from a subject with minimizednoise artifact distortion and used as a ground truth data set intraining a neural network to de-blur MR images with off-resonancecorrection procedures according to this disclosure.

FIG. 7B is a simulated field map of off-resonant frequencies within aselected frequency range that may be used to map a simulated blurredimage from the ground truth image of FIG. 7A.

FIG. 7C is a blurred frame of MR image data developed from the groundtruth image of FIG. 7A and subject to a simulated k-space trajectorymatched to the field map of FIG. 7B, such that the structural similaritymetric (SSIM) for the blurred image, compared to the ground truth imageof FIG. 7A is equal to 0.7965, and the peak signal to noise ratio (PSNR)for the blurred image, compared to the ground truth image of FIG. 7A, isequal to 27.79.

FIG. 7D is a frame of MR image data that has been de-blurred accordingto procedures and convolutional neural networks set forth in thisdisclosure, wherein the de-blurred image is a convolutional neuralnetwork output that shows improvements in structural similarity metric(SSIM) at 0.9611 and peak signal to noise ratio (PSNR) at 38.14 ascompared to the blurred image of FIG. 7C and when compared to the groundtruth image data of FIG. 7A.

FIG. 7E is a frame of MR image data representing the differences inpixel values in image space for the blurred image of FIG. 7C compared tothe ground truth image of FIG. 7A.

FIG. 7F is a frame of MR image data representing the differences, whichapproach zero, in pixel values in image space for the convolutionalneural network output of a de-blurred image of FIG. 7D compared to theground truth image of FIG. 7A.

FIG. 8A illustrates an overall frame of blurred MR image data and aclose up section of the frame of image data, wherein the imagecorresponds to a spiral scan of a standardized phantom object used totest and calibrate the off-resonance correction methods implemented inthis disclosure.

FIG. 8B illustrates an overall frame of corrected MR image data and aclose up section of the frame of corrected MR image data, wherein theimage corresponds to the standardized phantom object of FIG. 8A used totest and calibrate the off-resonance correction methods implemented inthis disclosure and has been corrected with off-resonant de-blurringtechniques disclosed herein.

FIG. 8C illustrates an overall frame of corrected MR image data and aclose up section of the frame of corrected MR image data, wherein theimage corresponds to a standardized phantom object used to test andcalibrate the off-resonance correction methods implemented in thisdisclosure and has been corrected with a semi-automated off-resonantde-blurring techniques that also use a low resolution field map foradditional correction in a semi-automatic system.

FIG. 9A illustrates an overall frame of blurred MR image data and aclose-up section of the frame of image data, wherein the imagecorresponds to a head image of a human using a spiral scan with SPAMM(SPAtial Modulation of Magnetization) which is a technique whereRF-saturation pulses are used to place stripes or grids on the image fortagging to aid visualization of the blurring.

FIG. 9B illustrates an overall frame of corrected MR image data and aclose up section of the frame of corrected MR image data using the CNNof this disclosure, wherein the the image corresponds to a head image ofa human using a spiral scan with SPAMM (SPAtial Modulation ofMagnetization) which is a technique where RF-saturation pulses are usedto place stripes or grids on the image for tagging to aid visualizationof the blurring.

FIG. 9C illustrates an overall frame of corrected MR image data and aclose up section of the frame of corrected MR image data, wherein theimage corresponds to a head image of a human using a spiral scan withSPAMM (SPAtial Modulation of Magnetization) which is a technique whereRF-saturation pulses are used to place stripes or grids on the image fortagging to aid visualization of the blurring. The image has beencorrected with a semi-automated off-resonant de-blurring techniques thatalso use a low resolution field map for additional correction in asemi-automatic system.

DETAILED DESCRIPTION

Although example embodiments of the present disclosure are explained indetail herein, it is to be understood that other embodiments arecontemplated. Accordingly, it is not intended that the presentdisclosure be limited in its scope to the details of construction andarrangement of components set forth in the following description orillustrated in the drawings. The present disclosure is capable of otherembodiments and of being practiced or carried out in various ways.

It must also be noted that, as used in the specification and theappended claims, the singular forms “a,” “an” and “the” include pluralreferents unless the context clearly dictates otherwise. Ranges may beexpressed herein as from “about” or “approximately” one particular valueand/or to “about” or “approximately” another particular value. When sucha range is expressed, other exemplary embodiments include from the oneparticular value and/or to the other particular value.

By “comprising” or “containing” or “including” is meant that at leastthe named compound, element, particle, or method step is present in thecomposition or article or method, but does not exclude the presence ofother compounds, materials, particles, method steps, even if the othersuch compounds, material, particles, method steps have the same functionas what is named.

In describing example embodiments, terminology will be resorted to forthe sake of clarity. It is intended that each term contemplates itsbroadest meaning as understood by those skilled in the art and includesall technical equivalents that operate in a similar manner to accomplisha similar purpose. It is also to be understood that the mention of oneor more steps of a method does not preclude the presence of additionalmethod steps or intervening method steps between those steps expresslyidentified. Steps of a method may be performed in a different order thanthose described herein without departing from the scope of the presentdisclosure. Similarly, it is also to be understood that the mention ofone or more components in a device or system does not preclude thepresence of additional components or intervening components betweenthose components expressly identified.

As discussed herein, a “subject” (or “patient”) may be any applicablehuman, animal, or other organism, living or dead, or other biological ormolecular structure or chemical environment, and may relate toparticular components of the subject, for instance specific organs,tissues, or fluids of a subject, may be in a particular location of thesubject, referred to herein as an “area of interest” or a “region ofinterest.”

Some references, which may include various patents, patent applications,and publications, are cited in reference lists and discussed in thedisclosure provided herein. The citation and/or discussion of suchreferences is provided merely to clarify the description of the presentdisclosure and is not an admission that any such reference is “priorart” to any aspects of the present disclosure described herein. In termsof notation, “[n]” corresponds to the n^(th) reference in the list. Forexample, “[3]” refers to the 3^(rd) reference in the list, namely Zhang,K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a Gaussian Denoiser:Residual Learning of Deep CNN for Image Denoising. IEEE Transactions onImage Processing. 26, 3142-3155 (2017). All references cited anddiscussed in this specification are incorporated herein by reference intheir entireties and to the same extent as if each reference wasindividually incorporated by reference.

A detailed description of aspects of the present disclosure, inaccordance with various example embodiments, will now be provided withreference to the accompanying drawings. The drawings form a part hereofand show, by way of illustration, specific embodiments and examples. Inreferring to the drawings, like numerals represent like elementsthroughout the several figures. Some experimental data are presentedherein for purposes of illustration and should not be construed aslimiting the scope of the present disclosure in any way or excluding anyalternative or additional embodiments.

Reference [15] below (Zbontar et al., 2019) offers a comprehensiveoverview of MR imaging. The Zbontar reference [9B] is incorporated byreference herein. Zbontar explains how a fastMRI dataset, which is acollection of both raw MR measurements and images collected fromclinicians, is available for research purposes. The research supportingthis disclosure has utilized such a database to create, train, test anduse the de-blurring techniques described herein.

Zbontar includes a useful overview of MR imaging and states “[d]uringimaging, a sequence of spatially and temporally varying magnetic fields,called a pulse sequence, is applied by the MRI machine. This induces thebody to emit resonant electromagnetic response fields which are measuredby the receiver coil. The measurements typically correspond to pointsalong a prescribed path through the multi-dimensional Fourier-spacerepresentation of an imaged body. This Fourier space is known as k-spacein the medical imaging community. In the most basic usage of MR imaging,the full Fourier-space representation of a region is captured by asequence of samples that tile the space up to a specified maximumfrequency. The spatially-resolved image m can be estimated from the fullk-space y by performing an inverse multidimensional Fourier transform.”Zbontar, ref. [15], pages 2-3. The fastMRl Dataset referenced in theZbontar reference [15] includes vast collections of this kind of MRimage data for research use. Zbontar explains that the data can be usedfor MRI single-coil and multi-coil reconstructions. As their nameimplies these kinds of reconstructions can approximate a respectiveground truth image from under-sampled single coil data or under-sampledmulti-coil data, where under-sampling helps with processing times.According to Zbontar, ground truth images are “real-valued imagesreconstructed from fully-sampled multi-coil acquisitions.” Zbontar, page6. These aspects of the Zbontar reference [15] are utilized furtherherein and are provided as a brief introduction to the subject matter ofthis disclosure.

Additional background information is provided in FIGS. 1-4 . FIG. 1 is asystem diagram illustrating an operating environment capable ofimplementing aspects of the present disclosure in accordance with one ormore example embodiments. FIG. 1 illustrates an example of a magneticresonance imaging (MRI) system 100, including a data acquisition anddisplay computer 150 coupled to an operator console 110, an MRIreal-time control sequencer 152, and an MRI subsystem 154. The MRIsubsystem 154 may include XYZ magnetic gradient coils and associatedamplifiers 168, a static Z-axis magnet 169, a digital RF transmitter162, a digital RF receiver 160, a transmit/receive switch 164, and RFcoil(s) 166. The MRI subsystem 154 may be controlled in real time bycontrol sequencer 152 to generate magnetic and radio frequency fieldsthat stimulate magnetic resonance phenomena in a living subject, patientP, to be imaged. A contrast-enhanced image of an area of interest A ofthe patient P may be shown on display 158. The display 158 may beimplemented through a variety of output interfaces, including a monitor,printer, or data storage.

The area of interest “A” corresponds to a region associated with one ormore physiological activities in patient “P”. The area of interest shownin the example embodiment of FIG. 1 corresponds to a chest region ofpatient “P”, but the area of interest for purposes of implementingaspects of the disclosure presented herein is not limited to the chestarea. It should be recognized and appreciated that the area of interestcan be one or more of a brain region, heart region, and upper or lowerlimb regions of the patient “P”, for example.

It should be appreciated that any number and type of computer-basedmedical imaging systems or components, including various types ofcommercially available medical imaging systems and components, may beused to practice certain aspects of the present disclosure. Systems asdescribed herein with respect to example embodiments are not intended tobe specifically limited to magnetic resonance imaging (MRI)implementations or the particular system shown in FIG. 1 .

One or more data acquisition or data collection steps as describedherein in accordance with one or more embodiments may include acquiring,collecting, receiving, or otherwise obtaining data such as imaging datacorresponding to an area of interest. By way of example, dataacquisition or collection may include acquiring data via a dataacquisition device, receiving data from an on-site or off-site dataacquisition device or from another data collection, storage, orprocessing device. Similarly, data acquisition or data collectiondevices of a system in accordance with one or more embodiments of thepresent disclosure may include any device configured to acquire,collect, or otherwise obtain data, or to receive data from a dataacquisition device within the system, an independent data acquisitiondevice located on-site or off-site, or another data collection, storage,or processing device.

FIG. 2 is a computer architecture diagram showing a general computingsystem capable of implementing aspects of the present disclosure inaccordance with one or more embodiments described herein. A computer 200may be configured to perform one or more functions associated withembodiments of this disclosure. For example, the computer 200 may beconfigured to perform operations for denoising MR images as describedherein with respect to certain embodiments. It should be appreciatedthat the computer 200 may be implemented within a single computingdevice or a computing system formed with multiple connected computingdevices. The computer 200 may be configured to perform variousdistributed computing tasks, which may distribute processing and/orstorage resources among the multiple devices. The data acquisition anddisplay computer 150 and/or operator console 110 of the system shown inFIG. 1 may include one or more systems and components of the computer200.

As shown, the computer 200 includes a processing unit 202 (“CPU”), asystem memory 204, and a system bus 206 that couples the memory 204 tothe CPU 202. The computer 200 further includes a mass storage device 212for storing program modules 214. The program modules 214 may be operableto perform one or more functions associated with embodiments of methodas illustrated in one or more of the figures of this disclosure, forexample to cause the computer 200 to perform operations of the presentdisclosure as described below. The program modules 214 may include animaging application 218 for performing data acquisition functions asdescribed herein, for example to receive image data corresponding tomagnetic resonance imaging of an area of interest. The computer 200 caninclude a data store 220 for storing data that may includeimaging-related data 222 such as acquired image data, and a modelingdata store 224 for storing image modeling data, or other various typesof data utilized in practicing aspects of the present disclosure.

The mass storage device 212 is connected to the CPU 202 through a massstorage controller (not shown) connected to the bus 206. The massstorage device 212 and its associated computer-storage media providenon-volatile storage for the computer 200. Although the description ofcomputer-storage media contained herein refers to a mass storage device,such as a hard disk or CD-ROM drive, it should be appreciated by thoseskilled in the art that computer-storage media can be any availablecomputer storage media that can be accessed by the computer 200.

By way of example, and not limitation, computer-storage media (alsoreferred to herein as a “computer-readable storage medium” or“computer-readable storage media”) may include volatile andnon-volatile, removable and non-removable media implemented in anymethod or technology for storage of information such as computer-storageinstructions, data structures, program modules, or other data. Forexample, computer storage media includes, but is not limited to, RAM,ROM, EPROM, EEPROM, flash memory or other solid state memory technology,CD-ROM, digital versatile disks (“DVD”), HD-DVD, BLU-RAY, or otheroptical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other medium which canbe used to store the desired information and which can be accessed bythe computer 200. Transitory signals are not “computer-storage media”,“computer-readable storage medium” or “computer-readable storage media”as described herein.

According to various embodiments, the computer 200 may operate in anetworked environment using connections to other local or remotecomputers through a network 216 via a network interface unit 210connected to the bus 206. The network interface unit 210 may facilitateconnection of the computing device inputs and outputs to one or moresuitable networks and/or connections such as a local area network (LAN),a wide area network (WAN), the Internet, a cellular network, a radiofrequency network, a Bluetooth-enabled network, a Wi-Fi enabled network,a satellite-based network, or other wired and/or wireless networks forcommunication with external devices and/or systems. The computer 200 mayalso include an input/output controller 208 for receiving and processinginput from a number of input devices. Input devices may include one ormore of keyboards, mice, stylus, touchscreens, microphones, audiocapturing devices, or image/video capturing devices. An end user mayutilize such input devices to interact with a user interface, forexample a graphical user interface, for managing various functionsperformed by the computer 200.

The bus 206 may enable the processing unit 202 to read code and/or datato/from the mass storage device 212 or other computer-storage media. Thecomputer-storage media may represent apparatus in the form of storageelements that are implemented using any suitable technology, includingbut not limited to semiconductors, magnetic materials, optics, or thelike. The computer-storage media may represent memory components,whether characterized as RAM, ROM, flash, or other types of technology.The computer-storage media may also represent secondary storage, whetherimplemented as hard drives or otherwise. Hard drive implementations maybe characterized as solid state or may include rotating media storingmagnetically-encoded information. The program modules 214, which includethe imaging application 218, may include instructions that, when loadedinto the processing unit 202 and executed, cause the computer 200 toprovide functions associated with embodiments illustrated herein. Theprogram modules 214 may also provide various tools or techniques bywhich the computer 200 may participate within the overall systems oroperating environments using the components, flows, and data structuresdiscussed throughout this description.

In general, the program modules 214 may, when loaded into the processingunit 202 and executed, transform the processing unit 202 and the overallcomputer 200 from a general-purpose computing system into aspecial-purpose computing system. The processing unit 202 may beconstructed from any number of transistors or other discrete circuitelements, which may individually or collectively assume any number ofstates. More specifically, the processing unit 202 may operate as afinite-state machine, in response to executable instructions containedwithin the program modules 214. These computer-executable instructionsmay transform the processing unit 202 by specifying how the processingunit 202 transitions between states, thereby transforming thetransistors or other discrete hardware elements constituting theprocessing unit 202.

Encoding the program modules 214 may also transform the physicalstructure of the computer-storage media. The specific transformation ofphysical structure may depend on various factors, in differentimplementations of this description. Examples of such factors mayinclude but are not limited to the technology used to implement thecomputer-storage media, whether the computer storage media arecharacterized as primary or secondary storage, and the like. Forexample, if the computer-storage media are implemented assemiconductor-based memory, the program modules 214 may transform thephysical state of the semiconductor memory, when the software is encodedtherein. For example, the program modules 214 may transform the state oftransistors, capacitors, or other discrete circuit elements constitutingthe semiconductor memory.

As another example, the computer-storage media may be implemented usingmagnetic or optical technology. In such implementations, the programmodules 214 may transform the physical state of magnetic or opticalmedia, when the software is encoded therein. These transformations mayinclude altering the magnetic characteristics of particular locationswithin given magnetic media. These transformations may also includealtering the physical features or characteristics of particularlocations within given optical media, to change the opticalcharacteristics of those locations. Other transformations of physicalmedia are possible without departing from the scope of the presentdescription, with the foregoing examples provided only to facilitatethis discussion.

Commercially available medical imaging systems and components, may beused to practice certain aspects of the present disclosure. Thesecommercially available imaging systems include 1.5 T and 3 T MRIscanners.

FIG. 3A has been used previously in U.S. Pat. Pub. No. 2022/0188602(Meyer), which is incorporated by reference in its entirety. Withreference to background FIG. 3A, a schematic of a U-Net convolutionalneural network is shown. The boxes represent feature maps, and thenumber of channels is labelled on each feature map. Neural networkoperations set forth in the legend of FIG. 3A, including convolution andmax pooling, are represented by arrows between each feature map.

Some embodiments of the present disclosure may also include anUnsupervised Deep Convolutional Neural Network (U-DCNN) with structuralimprovements specifically for denoising MRIs. Some embodiments of thepresent disclosure only require a noisy MRI to be denoised as the inputand functions as a traditional filter so that no simulated or acquiredhigh-quality training data is needed. Instead of relying on averaging,the U-DCNN* uses a DCNN structure and therefore is more robust indenoising performances and maintaining fine structures, especially fornon-uniform noise in a clinical MR image. Embodiments of the presentdisclosure include different network designs with a variety of inputimages, network depths, and skip-connections.

The structure and hyper-parameters of the U-DCNN can be optimized forbrain MRI. Embodiments of the present disclosure have been validatedwith a simulated brain MRI dataset at various noise levels and anacquired dataset with parallel imaging. Comparisons with non-local means(NLM) and block-matching and 3D filtering (BM3D) were made,demonstrating a superior and more robust performance over thetraditional filter-based methods, especially on the acquired MRI withnon-uniform noise.

According to some embodiments of the present disclosure, the U-DCNN is adeep generator network, which can be regarded as a highly non-linearparametric function x=f _(θ)(z) that maps an input z to a denoised imagex. The parameters θ can be comprised of the weights and bias of thenetwork's filtering operations including convolutions, up-sampling andnon-linear activation functions. The final set of parameters can beobtained using an optimizer such as gradient descent and a lossfunction, starting from a random parameter initialization. As discussedin [4], such a network structure has high impedance to noise and lowimpedance to signal. In other words, when generating x, it is mucheasier to obtain the parameter set for an image than random noise, asthe patterns in an image can make the generation process moreconvenient. For natural images, U-DCNN 600 has demonstrated a fasterconvergence towards naturally-looking images than corrupted noisy images[4].

The denoising performance can be different between MRI and naturalimages because: 1) MRI has different image characteristics, 2) the finestructural details, such as small lesions, matter more in MRI, and 3)MRI noise is usually more complex than the uniform Gaussian noise onnatural images, especially with multiple receiver coils. In order tostudy the denoising capability of U-DCNN on MRI, the generation processfor the synthetic noise-free brain MRI, Rician noise itself and noisyMRI using the mean squared error (MSE) between the output of U-DCNN andthe specific target was examined.

According to some embodiments of the present disclosure, the method fordenoising images includes acquiring MR image data 160 of the area ofinterest A of the subject P and processing that data using a U-DCNN toremove the noise. The area of interest A may include at least a part ofthe brain of the subject P or patient. Embodiments of the presentdisclosure may be used to denoise images produced by magnetic resonanceangiography, diffusion MRI, perfusion MRI, or other medical imagingtechniques. The noise data can comprise non-uniform noise originatingfrom coils 168 used in multi-band MR image acquisitions. The noisy inputimages input to the system can be previously calculated as diagnosticcompilations of acquired image data from parallel channels, and thediagnostic compilations can include calculated images showing thesubject P or patient's apparent diffusion coefficient, cerebral bloodflow and cerebral blood volume. The high SNR image data can include MRimage data 160 acquired during different MRI sequences, including theT1, T2, and PD sequences.

Noisy input images can include noise data and noise-free image data.According to some embodiments of the present disclosure, the noisy inputimages are processed by running iterations of a converging sequence inthe U-DCNN, and updating the parameter settings used in calculating aseries of image feature sets with the U-DCNN. The parameter settings canbe updated in each iteration of the converging sequence. The convergingsequence of the U-DCNN can be terminated before the feature sets predictan output image that replicates the noise data from the noisy inputimage. According to some embodiments of the present disclosure, adenoised MR image of the area of interest A can be output based on theselected features set.

Embodiments of the present disclosure may be applied to 2D images, 3Dimages, or both. Embodiments of the present disclosure applied to 3Dimages can use the additional spatial information from the through-planedimension that is not present in a 2D image. The acquired MR image data160 may include multi-slice or 3D acquisition. For example, a slice-wisesliding window technique using 3D convolutions can be used. For brainMRI with multi-slice 2D or 3D acquisitions, the spatial informationalong the through-plane dimension can be integrated to improve theperformance [16]. Replacing the 2D convolutions with 3D convolutions inthe unsupervised DCNN can change it to a 3D network. According to someembodiments of the present disclosure, the network can take the entire3D stack as an input. However, according to some embodiments of thepresent disclosure, a slice-wise sliding window can be used byreconstructing a small number of slices (e.g. 8) together and sliding tothe next stack when one stack finishes denoising. Using a slice-wisesliding window method can avoid the greatly increased computation andmemory requirements of a network that takes the entire 3D stack as asingle input. The network structure on the slice dimension will also begreatly simplified to limit the extra computations. To furtheraccelerate the algorithm, which will become more of a bottleneck

when there are a large number of slices in one scan, ShuffleNet [17] canbe used. ShuffleNet divides the convolutions along the feature dimensioninto smaller groups and performs a two-step process to first runconvolutions within groups and then summarizes the output from differentgroups. ShuffleNet has shown advantages in computation speed withminimal or no loss of accuracy.

As the unsupervised DCNN is used for denoising, the requirement forcollecting a large dataset for training and validation is alleviated.However, a decent-sized validation dataset that uses a variety ofsequences and acquisition strategies is still necessary. To evaluate thealgorithm against a noise-free gold standard and compare with othermethods, an open source simulated brain database (BrainWeb) [34A-38A]that includes T1, T2 and PD weighted images at a variety of slicethicknesses, noise levels, and levels of intensity non-uniformity may beused. It also includes both a normal brain and a brain with MS lesions.

FIG. 3B has been used previously in U.S. Pat. Pub. No. 2022/0373630(Dou), which is incorporated by reference in its entirety. Withreference to FIG. 3B, a method 300 for training a neural network tocorrect motion-induced artifacts in magnetic resonance images is shown.

At step 302, the method can include acquiring original spirally-sampledframes of motion-free magnetic resonance image (MRI) data of a targetobject. The images can be images that were acquired using any method,including using both conventional MRI sampling and spiral MIR sampling.

In some embodiments of the present disclosure, the method 300 can alsoinclude augmenting the original frames of motion free MRI data to formaugmented frames of motion free MRI data in the image domain. Theaugmentation can include applying different transforms to the originalframes. Non-limiting examples of transforms that can be applied inembodiments of the present disclosure include applying in-planerotations, horizontal flips, and/or vertical flips to the originalframes. In some embodiments of the present disclosure, the augmentedframes of motion-free MRI data and the respectively updated frames ofmotion-corrupted MRI data can be saved in a computer. As a non-limitingexample, the augmented frames and the respectively updated frames can besaved in the image domain format.

At step 304, a spatial transformation matrix can be applied to theoriginal frames of the motion-free MRI data to produce multiple framesof spiral MRI data having respective motion states.

At step 306, a non-uniform Fast Fourier Transform (NUFFT) can be appliedto each of the multiple frames of spiral MRI data having respectivemotion states to generate respective k-space data sets corresponding toeach of the multiple frames of spiral MRI data having respective motionstates.

At step 308, the respective k-space data sets can be combined. Thecombination of the respective K-space datasets can produce amotion-corrupted k-space data set of spiral MRI data.

At step 310, an adjoint NUFFT can be applied to the motion-corruptedk-space data set and respectively updated frames of motion-corrupted MRIdata in the image domain can be formed.

At step 312, a neural network can be trained that generates outputframes of motion free MRI data using the respectively updated frames ofmotion corrupted MRI data. In some embodiments of the presentdisclosure, step 312 can include training a generative adversarialnetwork with augmented frames of motion-free MRI data and therespectively updated frames of motion-corrupted MRI data. As anon-limiting example, training the generative adversarial network caninclude applying the respectively updated frames of motion-corrupted MRIdata to a generator in the generative adversarial network to producerespective motion compensated images accessible by a discriminator inthe generative adversarial network. In some embodiments of the presentdisclosure, training the generative adversarial network can also includeapplying the respectively updated frames of motion-corrupted MRI dataand the respective motion compensated images to a discriminator withinthe generative adversarial network.

In some embodiments of the present disclosure, the generativeadversarial network can be trained by applying the respectively updatedframes of motion-corrupted MRI data and a target motion-free image to adiscriminator within the generative adversarial network. The generativeadversarial network that can be trained in step 312 can also be trainedto minimize or maximize a function, for example a function related toimage quality. As a non-limiting example the function can be an errorfunction and the system can be configured to minimize the errorfunction. A non-limiting example of an error function is a function thatrepresents the amount of error in the output images.

At step 314, the trained neural network model can be saved. The trainedneural network model can correspond to corrections applicable to theupdated frames of motion corrupted MRI data that generate the outputframes of motion-free MRI data.

FIG. 3B illustrates another method 350 according to an embodiment of thepresent disclosure. The method 350 can train a neural network to correctmotion-induced errors in magnetic resonance images.

At step 352 original frames of motion-free magnetic resonance image(MRI) data of a target object can be acquired.

At step 354, spiral interleaves for spatial transformation can beselected for each original frame of motion-free MRI data, and arespective spatial transformation matrix is applied to the selectedspiral interleaves therein to produce multiple frames of spiral MRI datahaving respective motion states.

In some embodiments of the present disclosure, step 354 can includeselecting spiral interleaves for spatial transformation by dividing allspiral interleaves within the original frames into a selected number ofsets, wherein each set is subject to a respective motion eventcorresponding to a respective spatial transformation matrix.

In some embodiments of the present disclosure, applying the spatialtransformation can include simulating in plane rigid motion artifactsfrom the original frames to produce the multiple frames of spiral MRIdata having respective motion states.

In some embodiments of the present disclosure, the number of spiralinterleaves in a set can randomly selected from {8, 16, 32, 64, 128}.Additionally, in some embodiments, the spatial transformation of thespiral interleaves can include a type of spiral trajectory randomlyselected from constant density, variable density, and dual densitytransformations.

At step 356, a Non-uniform Fast Fourier Transform (NUFFT) can be appliedto each of the multiple frames of spiral MRI data having respectivemotion states to generate respective k-space data sets corresponding toeach of the multiple frames of spiral MRI data having respective motionstates;

At step 358, the respective k-space data sets are combined to produce amotion-corrupted k-space data set of spiral MRI data.

At step 360, an adjoint NUFFT can be applied to the motion-corruptedk-space data set and respectively updated frames are formed ofmotion-corrupted MRI data in the image domain. In some embodiments ofthe present disclosure, forming the multiple frames of spiral MRI datahaving respective motion states can include applying in-plane horizontaland vertical translations and/or in plane rotations to the originalframes of motion free MRI data.

At step 362, a neural network can be trained that generates outputframes of motion free MRI data using the respectively updated frames ofmotion corrupted MRI data that generate the output frames of motion freeMRI data. At step 364, data can be saved corresponding to correctionsapplicable to the updated frames of motion corrupted MRI data togenerate the output frames of motion free MRI data.

Some methods of MR image data correction include a computer-implementedmethod of training a neural network to correct motion-induced errors inmagnetic resonance images by acquiring original frames of motion freemagnetic resonance image (MRI) data of a target object. Fouriertransforms are used to acquire respective original k-space data setscorresponding to each original frame. The method continues by applying arespective spatial transformation matrix to each original k-space dataset to acquire motion state data for each original k-space data set. Thecomputer then replaces portions of each original k-space data set withthe motion state data to produce a transformed k-space MRI data sethaving a respective motion state and by combining the transformedk-space MRI data sets, the method produces a motion-corrupted k-spacedata set of MRI data. To return back to the image domain, the methodincludes applying an inverse Fourier transform to the motion-corruptedk-space data set and forming respective synthetic motion corruptedframes of MRI data. The synthetic motion corrupted frames of MRI dataare used to train a neural network that generates output frames ofmotion compensated MRI data. Applying the spatial transformation matrixincludes simulating in plane rigid motion artifacts from the originalframes to produce the multiple frames of MRI data having the respectivemotion states. In some non-limiting embodiments, the Fourier transformis a Fast Fourier Transform and the k-space data sets are Cartesiank-space data sets.

In other related systems, the Fourier transform is a Non-Uniform FourierTransform and the k-space data sets are formed on spiral k-spacetrajectories. Replacing portions of each original k-space data setfurther includes replacing selected spiral interleaves in a respectivek-space data set. Replacing selected spiral interleaves may includedividing all spiral interleaves in the motion-corrupted k-space data setinto a selected number of sets, wherein each set is subject to arespective motion event corresponding to a respective spatialtransformation matrix. In non-limiting embodiments, a number of spiralinterleaves in each of the sets is randomly selected from a group ofnumbers including 8, 16, 32, 64, and 128. A spatial transformation ofthe spiral interleaves may include a type of spiral trajectory randomlyselected from constant density, variable density, and dual densitytransformations. The number of sets is selected from 1, 2, 3, and 4sets.

In addition to the above noted use of convolutional neural networks,this disclosure includes using more advanced networks, particularlyde-noising convolutional neural networks as set forth in reference [23],which is incorporated herein by reference in its entirety. In thearticle of reference [23], Zhang et al. (“Zhang”) describes advances inde-noising convolutional neural networks with residual layerpredictions. Zhang uses feed-forward, denoising convolutional neuralnetworks (DnCNNs) to embrace the progress in very deep architecture,learning algorithms, and

regularization methods for image denoising. The DnCNN model of reference[23] is able to handle Gaussian denoising with unknown noise levels(i.e., blind Gaussian denoising) and even additive white Gaussian noise(AWGN).

Over the last few decades, various models have been exploited formodeling image priors in the context of de-noising algorithms, includingnonlocal self-similarity (NSS) models. To overcome the limitations ofprior-based approaches, several discriminative learning methods havebeen recently developed to learn image prior models in the context oftruncated inference

procedure. In Zhang's work of reference [23], instead of learning adiscriminative model with an explicit image prior, Zhang treats imagedenoising as a plain discriminative learning problem, i.e., separatingthe noise from a noisy image by feed-forward convolutional neuralnetworks (CNN). Zhang explains, at page 1, the reasons for using CNN asbeing “three-fold.” “First, CNN with very deep architecture is effectivein increasing the capacity and flexibility for exploiting imagecharacteristics. Second, considerable advances have been achieved onregularization and learning methods for training CNN, includingRectifier Linear Unit (ReLU), batch normalization and residual learning.These methods can be adopted in CNN to speed up the training process andimprove the denoising performance. Third, CNN is well-suited forparallel computation on modern powerful GPU, which can be exploited toimprove the run time performance. Rather than directly outputting thedenoised image Ax, the proposed DnCNN is designed to predict theresidual image {circumflex over ( )}v, i.e., the difference between thenoisy observation and the latent clean image. The proposed DnCNNimplicitly removes the latent clean image with the operations in thehidden layers.” See reference [23], Zhang, page 2.

Zhang contrasts “the existing deep neural network-based methods whichdirectly estimate the latent clean image with the Zhang network thatadopts the residual learning strategy to remove the latent clean imagefrom noisy observation. The residual network explicitly learns aresidual mapping for a few stacked layers. With such a residual learningstrategy, extremely deep CNN can be easily trained and improved accuracyhas been achieved for image classification and object detection.” Ref.[23], Zhang page 2.

FIG. 4 is a schematic of Zhang's network, which also shows that “batchnormalization is proposed to alleviate the internal covariate shift byincorporating a normalization step and a scale and shift step before thenonlinearity in each layer. For batch normalization, only two parametersper activation are added, and they can be updated withback-propagation.” Reference [23], Zhang page 3. Zhang describes thedeep architecture of FIG. 4 as a “DnCNN with depth D, [and] three typesof layers . . . . (i) Conv+ReLU: for the first layer, 64 filters of size3×3×c are used to generate 64 feature maps, and rectified linear units(ReLU, max(0; ●)) are then utilized for nonlinearity. Here c representsthe number of image channels, i.e., c=1 for gray image and c=3 for colorimage. (ii) Conv+BN+ReLU: for layers 2˜(D−1), 64 filters of size 3×3×64are used, and batch normalization is added between convolution and ReLU.(iii) Cony: for the last layer, c filters of size 3×3×64 are used toreconstruct the output.” Ref. [23], Zhang, page 5.

One non-limiting goal of this disclosure is to utilize the aboveconcepts, particularly the de-noising convolutional neural networks, butto also include calculations of complex image data that includes phaseinformation, instead of simply relying upon magnitude data or other realnumeric data that represents an image. This process not only requiresadjustments for the kind of data, but certain steps in the process, suchas batch normalization and rectified linear units, are also adjusted aspart of this disclosure.

The advancements in complex neural networks has also requiredadvancements in certain steps such as the rectified linear units usedtherein. A rectified linear unit is a process that ensures the imagedata remains positive during convolutions. The ReLU outputs the samevalue for a positive input and a zero for a negative input to expeditecalculations during use of convolutional neural network. This disclosurequotes portions of reference [24] by Trabelsi, et al. (“Trabelsi”) forclarity in certain processes, including the ReLU application in acomplex domain. Trabelsi, page 2, indicates “the advantages of usingcomplex-valued representations with respect to retrieval and insertioninto an associative memory. In residual networks, the output of eachblock is added to the output history accumulated by summation until thatpoint.” Trabelsi notes that the work in reference [24] “incorporate[s]complex weights and activations in residual networks.”

Trabelsi indicates that “the phase component is not only important froma biological point of view but also from a signal processingperspective. It has been shown that the phase information in speechsignals affects their intelligibility (Shi et al., 2006). Also Oppenheimand Lim (1981) show that the amount of information present in the phaseof an image is sufficient to recover the majority of the informationencoded in its magnitude. In fact, phase provides a detailed descriptionof objects as it encodes shapes, edges, and orientations.” Going intodetail in the complex convolution theory, Trabelsi states that “[i]norder to perform the equivalent of a traditional real-valued 2Dconvolution in the complex domain, [Trabelsi] convolve[s] a complexfilter matrix W=A+iB by a complex vector h=x+iy where A and B are realmatrices and x and y are real vectors since this disclosure issimulating complex arithmetic using real-valued entities. As theconvolution operator is distributive, convolving the vector h by thefilter results in W*h=(A*x−B*y)+i(B*x+A*y).” Trabelsi, page 4. As partof Trabelsi's complex convolution, Trabelsi applies a complex rectifiedlinear unit (“ReLU”) on both of the real and imaginary part of a neuronwithin Trabelsi's system.

This disclosure incorporates by reference the entire disclosures of theabove discussed U.S. Pat. Pub. No. 2022/0188602 (Meyer), U.S. Pat. Pub.No. 2022/0373630 (Dou), and

the published technical journal articles by Zhang (reference [23]) andTrabelsi (reference [24]). Against the back-drop of these articles andothers incorporated by reference below, this disclosure adds significantimprovements to all aspects of using de-noising convolutional neuralnetworks (DnCNNs) within the domain of complex data, which would includephase information for collected image data. In general, MR imagesobtained at low-field inherently have low signal to noise ratios.

Off-resonance is a major limitation for spiral imaging. FIGS. 5-9Cillustrate embodiments of this disclosure by which MR image data can becorrected, or de-blurred, by using convolutional neural networksdesigned to correct off-resonance artifacts present in the images. Theoff-resonance portions of the image are noisy and incorporate blurredregions or other unwanted distortions in the frame of image data. Priortechniques to account for off-resonant image distortions have includeddeveloping complex field maps that track image data within the framescollected at off-resonant frequencies. Simple examples of creating afield map include calculating phase differences between images collectedwith different echo times. Once the phase differences and frequencyoffsets in a set of images are known, they can be accounted for bycorrection algorithms; however, relying upon field maps is complicatedbecause field maps are time and resource intensive to create. Thisdisclosure allows for correcting off resonant image blurring in MRI datacollection without relying upon field maps to de-blur the image. Thisdisclosure illustrates a convolutional neural network that wasimplemented in this study to correct off-resonance artifacts withoutfield maps. The network was trained on images with simulated blurringartifacts. The image quality was improved after the correction for bothsimulated data and in vivo data.

Spiral data sampling has several advantages, such as short echo time,high scan efficiency, and motion robustness. However, spiral imaging islimited by blurring artifacts raised from off-resonance effects,especially with long readouts. Most existing deblurring methods requireprior knowledge about the field map [25-28]. Automatic off-resonancecorrection methods do not require field maps, but their performance ishighly dependent on the choice of the objective function and frequencysearching range [29, 30]. Recently, convolutional neural networks (CNNs)demonstrated promising results for image deblurring [31, 32]. This workdiscloses and develops a convolutional neural network (CNN) to performautomatic off-resonance correction for spiral imaging. An open-sourcedata set (https://fastmri.med.nyu.edu/) containing Cartesian T2-weightedimages for 4179 subjects was used. Single-coil data was simulated fromthe multi-coil data using an emulated single-coil method (p. 2 “using alinear combination of the responses from multiple coils for the emulatedsingle coil (ESC) response”).

This work randomly selected 1000 subjects and divided the 1000 imagingvolumes such that 750 volumes were used for training and validation, andthe remaining 250 volumes were used for testing. To generate pairedtraining data with spiral off-resonance artifacts, a field map was firstsimulated by combining a random 2D polynomial with several random 2DGaussian functions. A brain mask was applied to the generated field mapto simulate the abrupt off-resonance change at the air-tissue boundary.The frequency range of the simulated field map was limited to between[−500 Hz, +500 Hz]. Then, a spiral k-space trajectory was simulatedbased on the image field of view (FOV) and resolution, and thecorresponding Cartesian k-space time map was calculated for fastercomputation. The number of spiral interleaves was randomly selectedbetween 4 to 64, and the readout length for one interleave was between 4ms and 24 ms.

The synthesized blurred image was obtained by applying the simulatedfield map to the ground truth through multifrequency interpolation (MFI)[26], as shown in FIG. 5. To correct the off-resonance artifacts, thiswork explains how to develop and train a CNN with three residual blocksthat are collectively labeled in the figures as AutofocusNet. Theblurred image was first demodulated at 11 different frequencies, whichis also the first step in the conventional autofocus method [29]. Theinput of the network was a series of the 11 images, with 22 channelscorresponding to real and imaginary components. The output was thedeblurred image, with 2 channels corresponding to real and imaginarycomponents. Each residual block consists of two 5×5 convolutional layerswith filter depths of 128, followed by rectified linear unit (ReLU)activations. A skip connection was added between the block input andoutput, as shown in FIG. 6 .

The network was implemented in PyTorch [36]. L1 loss, i.e., the absolutevalue differences, between the network output and ground truth wasoptimized using Adam [37] with a learning rate of 0.0001. To avoidoverfitting, this disclosure adopted random patch cropping as thetraining augmentation. The structural similarity index (SSIM) and peaksignal-to-noise ratio (PSNR) were calculated as image quality metrics toevaluate the network performance on the simulated dataset. The trainednetwork was also applied to a phantom image and a head image of ahealthy volunteer acquired on a Siemens Avanto 1.5T scanner. The spiralimaging parameters were FOV=28 cm², matrix size=512×512, number ofinterleaves=14, and readout length=16.4 ms. Semiautomatic correctionutilizing a low-resolution field map was also performed for comparison.

FIGS. 7A-7F show the network performance on an image with simulatedspiral off-resonance artifacts. Compared to the blurred image, thenetwork output showed less artifacts, sharper structures, and higherimage quality. Quantitatively, the average SSIM and PSNR of thesimulated dataset were increased from 0.7231 and 26.84 to 0.9365 and36.71 after off-resonance correction.

The network performance on a phantom image and an in-vivo head imagecollected with spiral scans were shown in FIGS. 8A-8C and FIGS. 9A-9Crespectively. The network effectively recovered the image sharpness andimproved the image quality in both cases. These results showed that thenetwork could correct off-resonance artifacts for images outside thetraining dataset population. The network removed blurring in bothsituations with an average processing time of 50 ms for a single sliceon a single GPU (not including the I/O time). The network can begeneralized for concomitant gradient correction.

Considering the FIGS. 5 through 9C in more detail, a computerimplemented method of denoising a magnetic resonance (MR) image includesacquiring complex magnetic resonance (MR) image data 525 of an area ofinterest of a subject, wherein the image data comprises complex blurredimages 550 of multi-coil MR image data, and wherein the complex blurredimages comprise resonant image data and off-resonance artifact data. Foreach of the complex blurred images 550, the method includes demodulating600A the complex blurred images at a selected number (n) of frequenciesto form, for each of the n frequencies, a respective real componentframe 608 of the MR data and a respective imaginary component frame 607of the MR data. This disclosure uses layered data sets 603 as inputs toa convolutional neural network operation 622, and the layered data sets603 are formed by stacking 620A, 620B the respective real componentframes 608 and the respective imaginary component frames 607. Thelayered data set may be used as an input to a convolutional neuralnetwork (CNN) having a plurality of residual blocks 635A, 635B, 635C,and the residual blocks may include multiple convolution calculations600B paired with respective skip connections 640A, 640B, 640C. After afinal 1×1 convolution 600C, the output from the CNN is a de-blurred realimage frame 655 and a de-blurred imaginary image frame 650 of the MRdata for each complex blurred image 603. In some embodiments, the methoddemodulates the complex blurred images at n=11 frequencies, such thatthe layered data set is a three dimensional data set having 2n number ofchannels (22) for use as the input to the CNN operations 622. In somenon-limiting examples, compiling the layered data set includes stackingalternating frames of the real component frames 608 and the imaginarycomponent frames 607 for each selected frequency at which a respectivecomplex blurred image has been demodulated.

As shown in FIG. 6 , the residual blocks 635A, 635B, 635C may includemultiple convolutional calculations 600B, a rectified linear unitactivation, and one of the respective skip connections 640A, 640B, 640C.The respective skip connections add a block input to a block output,corresponding to a respective residual block, to provide data continuitybetween earlier residual blocks and later residual blocks of the CNN. Innon-limiting embodiments, the residual blocks 635A, 635B, 635C have twosequential convolutions 610A of the layered data set 621. In somenon-limiting embodiments, the CNN may have three consecutive residualblocks of two sequential convolutions. Without limiting this disclosureand illustrated in FIG. 6 , the convolutional computations 600B include5×5 convolution layers with filter depths of 128 followed by rectifiedlinear unit activations. The method of FIG. 6 is particularly adept atminimizing L1 loss (absolute value differences) between an output of theCNN and a ground truth image.

As shown in FIG. 5 , this disclosure also includes training the CNNoperations 622 with a ground truth image 525 and a simulated blurredimage 550 having simulated blurring artifacts by forming the simulatedimage from an in-vivo ground truth image. In one non-limiting step, thesimulated blurred image 550 is created with a brain mask 515 based onthe ground truth image 525 to simulate off-resonance changes at anair-tissue interface boundary in the ground truth image. Prior to usingthe mask, the work includes forming the brain mask 525 by applyingthresholding techniques 517 to the ground truth image 525. Using Otsu'sthresholding technique and/or a convex hull thresholding technique arenon-limiting ways to create the brain mask. The brain mask 517 helps toemphasize the air-tissue interface where clarity in the image can behard to achieve.

Forming the simulated blurred image includes generating a simulatedfield map 520 by forming a combination of a random 2D polynomial 505with at least one random 2D Gaussian function 510 and applying thecombination to the brain mask 515. Next, this work generates thesimulated blurred image 550 by applying the simulated field map 520 tothe ground truth 525 with multifrequency interpolation, wherein themultifrequency interpolation utilizes a simulated k-space trajectory 535of the ground truth image 525. Random patch cropping and other trainingaugmentations may be incorporated herein for each simulated blurredimage. As noted above, the simulated image for training may start withmulti-coil data or other techniques include forming the simulated imagewith emulated single coil data based on the multi-coil data.

The simulated images with the blurring along with paired ground truthimages as shown are used to train the convolutional neural networks ofthis disclosure. The convolutional neural networks with the feed forwardarrangement of computations in each residual block are particularlyuseful in reducing off-resonant blurring in MR images.

One non-limiting theory of operation of the CNN operations 622 lies inthe alternating stacked input data 620A, 620B keeping all phase andmagnitude information together in the compilations of data. The outputshave shown to be productive as illustrated in FIGS. 7A through 9C.

FIG. 7A is a frame 705 of MR image data taken from a subject withminimized noise artifact distortion and used as a ground truth data setin training a neural network to de-blur MR images with off-resonancecorrection procedures according to this disclosure.

FIG. 7B is a simulated field map 707 of off-resonant frequencies withina selected frequency range that may be used to map a simulated blurredimage 709 from the ground truth image 705 of FIG. 7A.

FIG. 7C is a simulated blurred frame 709 of MR image data developed fromthe ground truth image 705 of FIG. 7A and subject to a simulated k-spacetrajectory 535 matched to the field map 707 of FIG. 7B, such that thestructural similarity metric (SSIM) for the blurred image, compared tothe ground truth image of FIG. 7A is equal to 0.7965, and the peaksignal to noise ratio (PSNR) for the blurred image, compared to theground truth image of FIG. 7A, is equal to 27.79.

FIG. 7D is an output frame 713 of MR image data that has been de-blurredaccording to procedures and convolutional neural networks set forth inthis disclosure, wherein the de-blurred image is a convolutional neuralnetwork output that shows improvements in structural similarity metric(SSIM) at 0.9611 and peak signal to noise ratio (PSNR) at 38.14 ascompared to the blurred image of FIG. 7C and when compared to the groundtruth image data of FIG. 7A.

FIG. 7E is a frame 711 of MR image data representing the differences inpixel values in image space for the blurred image 709 of FIG. 7Ccompared to the ground truth image 705 of FIG. 7A.

FIG. 7F is a frame 715 of MR image data representing the differences,which approach zero, in pixel values in image space for theconvolutional neural network output of a de-blurred image of FIG. 7Dcompared to the ground truth image of FIG. 7A.

FIG. 8A illustrates an overall frame 805A of blurred MR image data and aclose up section 805B of the frame of image data, wherein the imagecorresponds to a spiral scan of a standardized phantom object used totest and calibrate the off-resonance correction methods implemented inthis disclosure.

FIG. 8B illustrates an overall frame 815A of corrected MR image data anda close up section 815B of the frame of corrected MR image data, whereinthe image corresponds to the standardized phantom object of FIG. 8A usedto test and calibrate the off-resonance correction methods implementedin this disclosure and has been corrected with off-resonant de-blurringtechniques disclosed herein.

FIG. 8C illustrates an overall frame 825A of corrected MR image data anda close up section 825B of the frame of corrected MR image data, whereinthe image corresponds to a standardized phantom object used to test andcalibrate the off-resonance correction methods implemented in thisdisclosure and has been corrected with a semi-automated off-resonantde-blurring techniques that also use a low resolution field map foradditional correction in a semi-automatic system.

FIG. 9A illustrates an overall frame 905A of blurred MR image data and aclose-up section 905B of the frame of image data, wherein the imagecorresponds to a head image of a human using a spiral scan with SPAMM(SPAtial Modulation of Magnetization) which is a technique whereRF-saturation pulses are used to place stripes or grids on the image fortagging to aid visualization of the blurring.

FIG. 9B illustrates an overall frame 915A of corrected MR image data anda close up section 915B of the frame of corrected MR image data, whereinthe image corresponds to a head image of a human using a spiral scanwith SPAMM (SPAtial Modulation of Magnetization) which is a techniquewhere RF-saturation pulses are used to place stripes or grids on theimage for tagging to aid visualization of the blurring.

FIG. 9C illustrates an overall frame 925A of corrected MR image data anda close up section 925B of the frame of corrected MR image data, whereinthe image corresponds wherein the image corresponds to a head image of ahuman using a spiral scan with SPAMM (SPAtial Modulation ofMagnetization) which is a technique where RF-saturation pulses are usedto place stripes or grids on the image for tagging to aid visualizationof the blurring. The image has been corrected with a semi-automatedoff-resonant de-blurring techniques that also use a low resolution fieldmap for additional correction in a semi-automatic system.

CONCLUSION

The specific configurations, choice of materials and the size and shapeof various elements can be varied according to particular designspecifications or constraints requiring a system or method constructedaccording to the principles of the present disclosure. Such changes areintended to be embraced within the scope of the present disclosure. Thepresently disclosed embodiments, therefore, are considered in allrespects to be illustrative and not restrictive. The patentable scope ofcertain embodiments of the present disclosure is indicated by theappended claims, rather than the foregoing description.

The following references are all incorporated by reference into thisdisclosure as if the text of each is set forth in full.

LIST OF REFERENCES

[1] Haysteen I, Ohlhues A, Madsen K H, et al. Are Movement Artifacts inMagnetic Resonance Imaging a Real Problem?—A Narrative Review. FrontNeurol. 2017; 8:232. doi: 10.3389/fneur.2017.00232. PMCID: PMC5447676.

-   -   [2] Edwards A D, Arthurs O J. Paediatric MRI under sedation: is        it necessary? What is the evidence for the alternatives? Pediatr        Radiol. 2011; 41(11):1353-1364. doi: 10.1007/s00247-011-2147-7.    -   [3] McGibney G, Smith M R, Nichols S T, Crawley A. Quantitative        evaluation of several partial Fourier reconstruction algorithms        used in MRI. Magn Reson Med. 1993; 30(1):51-59.    -   [4] Pruessmann K P, Weiger M, Scheidegger M B, Boesiger P.        SENSE: sensitivity encoding for fast MRI. Magn Reson Med. 1999;        42(5):952-962.    -   [5] Griswold M A, Jakob P M, Heidemann R M, et al. Generalized        autocalibrating partially parallel acquisitions (GRAPPA). Magn        Reson Med. 2002; 47(6):1202-1210.    -   [6] Lustig M, Pauly J M. SPIRiT: Iterative self-consistent        parallel imaging reconstruction from arbitrary k- space. Magn        Reson Med. 2010; 64(2):451-471. doi: 10.1002/mrm.22428. PMCID:        PMC2925465.    -   [7] Lustig M, Donoho D, Pauly J M. Sparse MRI: The application        of compressed sensing for rapid MR imaging. Magn Reson Med.        2007; 58(6):1182-1195.    -   [8] Glover G H, Li T Q, Ress D. Image-based method for        retrospective correction of physiological motion effects in        fMRI: RETROICOR. Magn Reson Med. 2000; 44(1):162-167.    -   [9] Maclaren J, Herbst M, Speck O, Zaitsev M. Prospective motion        correction in brain imaging: a review. Magn Reson Med. 2013;        69(3):621-636. doi: 10.1002/mrm.24314.    -   [10] Buades, A., Coll, B., Morel, J.-M.: Non-Local Means        Denoising. Image Processing On Line. 1, (2011).    -   [11] Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image        denoising with block-matching and 3D filtering. Image        Processing: Algorithms and Systems, Neural Networks, and Machine        Learning. (2006).    -   [12] cicek, Ozgun et al., 3D U-Net: Learning Dense Volumetric        Segmentation from Sparse Annotation published in MICCAI 2016.    -   [13] Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a        Gaussian Denoiser: Residual Learning of Deep CNN for Image        Denoising. IEEE Transactions on Image Processing. 26, 3142-3155        (2017).    -   [14] Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep image prior.        arXiv Preprints. (2017).    -   [15] Zbontar J, Knoll F, Sriram A, et al. fastMRI: An Open        Dataset and Benchmarks for Accelerated MRI. 2018.        arXiv:1811.08839 [cs.CV].    -   [16] Maggioni M, Katkovnik V, Egiazarian K, Foi A. Nonlocal        transform-domain filter for volumetric data denoising and        reconstruction. IEEE Trans Image Process. 2013; 22(1):119-133.        doi: 10.1109/TIP.2012.2210725.    -   [17] Zhang X, Zhou X, Lin M, Sun J. ShuffleNet: An Extremely        Efficient Convolutional Neural Network for Mobile Devices. arXiv        Preprints. 2017. arXiv:1707.01083.    -   [18] http://www.bic.mni.mcgill.ca/brainweb/    -   [19] Cocosco C A, Kollokian V, Kwan R K, Evans A C. BrainWeb:        Online Interface to a 3D MRI Simulated Brain Database.        Neurolmage, 1997; 5(4):5425.    -   [20] Kwan R K, Evans A C, Pike G B. MRI simulation-based        evaluation of image-processing and classification methods. IEEE        Trans Med Imaging. 1999; 18(11):1085-1097.    -   [21] Kwan R K, Evans A C, Pike G B. An Extensible MRI Simulator        for Post-Processing Evaluation. Visualization in Biomedical        Computing (VBC'96). Lecture Notes in Computer Science,        vol. 1131. Springer-Verlag, 1996. 135-140.    -   [22] Collins D L, Zijdenbos A P, Kollokian V, et al. Design and        Construction of a Realistic Digital Brain Phantom. IEEE Trans        Med Imaging. 1998; 17(3):463-468.    -   [23] Zhang K, Zuo W, Chen Y, Meng D, Zhang L. Beyond a Gaussian        Denoiser: Residual Learning of Deep CNN for Image Denoising. 13        Aug. 2016. arXiv:1608.03981 [cs.CV].    -   [24] Trabelsi C, Bilaniuk O, Zhang Y, et al. Deep Complex        Networks. 25 Feb. 2018. arXiv:1705.09792 [cs.NE].    -   [25] Noll D C, Meyer C H, Pauly J M, Nishimura D G, Macovski A.        A homogeneity correction method for magnetic resonance imaging        with time-varying gradients. IEEE Trans Med Imaging. 1991;        10(4):629-37.    -   [26] Man L C, Pauly J M, Macovski A. Multifrequency        interpolation for fast off-resonance correction. Magn Reson Med.        1997 May; 37(5):785-92.    -   [27] Ahunbay E, Pipe J G. Rapid method for deblurring spiral MR        images. Magn Reson Med. 2000 September; 44(3):491-4.    -   [28] Sutton B P, Noll D C, Fessler J A. Fast, iterative image        reconstruction for MRI in the presence of field inhomogeneities.        IEEE Trans Med Imaging. 2003 February; 22(2):178-88.    -   [29] Noll D C, Pauly J M, Meyer C H, Nishimura D G, Macovski A.        Deblurring for non-2D Fourier transform magnetic resonance        imaging. Magn Reson Med. 1992 June; 25(2):319-33.    -   [30] Man L C, Pauly J M, Macovski A. Improved automatic        off-resonance correction without a field map in spiral imaging.        Magn Reson Med. 1997 June; 37(6):906-13.    -   [31] Zeng D Y, Shaikh J, Holmes S, Brunsing R L, Pauly J M,        Nishimura D G, Vasanawala S S, Cheng J Y. Deep residual network        for off-resonance artifact correction with application to        pediatric body MRA with 3D cones. Magn Reson Med. 2019 October;        82(4):1398-1411.    -   [32] Lim Y, Bliesener Y, Narayanan S, Nayak K S. Deblurring for        spiral real-time MRI using convolutional neural networks. Magn        Reson Med. 2020 December; 84(6):3438-3452.    -   [33] Zbontar J, Knoll F, Sriram A, et al. fastMRI: An Open        Dataset and Benchmarks for Accelerated MRI. 2018.        arXiv:1811.08839 [cs.CV].    -   [34] Zbontar J, Tygert M. Simulating single-coil MRI from the        responses of multiple coils. 2018. arXiv:1811.08026 [eess.IV].    -   [35] He K, Zhang X, Ren S, Sun J. Deep residual learning for        image recognition. 2015. arXiv:1512.03385 [cs.CV].    -   [36] Paszke A, Gross S, Massa F, et al. PyTorch: An Imperative        Style, High-Performance Deep Learning Library. 2019.        arXiv:1912.01703 [cs.LG].    -   [37] Kingma D P, Ba J. Adam: A Method for Stochastic        Optimization. 2014. arXiv:1412.6980 [cs.LG].    -   [38] Chen W, Meyer C H. Semiautomatic off-resonance correction        in spiral imaging. Magn Reson Med. 2008 May; 59(5):1212-9.

1. A computer-implemented method of denoising a magnetic resonance (MR)image, comprising: acquiring complex magnetic resonance (MR) image dataof an area of interest of a subject, wherein the image data comprisescomplex blurred images of multi-coil MR image data, and wherein thecomplex blurred images comprise resonant image data and off-resonanceartifact data; for each of the complex blurred images: demodulating thecomplex blurred images at a selected number (n) of frequencies to form,for each of the n frequencies, a respective real component frame of theMR data and a respective imaginary component frame of the MR data;compiling a layered data set by stacking the respective real componentframes and the respective imaginary component frames; using the layereddata set as an input to a convolutional neural network (CNN) comprisinga plurality of residual blocks, wherein the residual blocks comprisemultiple convolution calculations paired with respective skipconnections; and outputting from the CNN a de-blurred real image frameand a de-blurred imaginary image frame of the MR data for each complexblurred image.
 2. The method of claim 1, further comprising:demodulating the complex blurred images at n=11 frequencies.
 3. Themethod of claim 1, further comprising: compiling the layered data set asa three dimensional data set having 2n number of channels for use as theinput to the CNN.
 4. The method of claim 1, further comprising:compiling the layered data set as alternating frames of the realcomponent frames and the imaginary component frames for each selectedfrequency at which a respective complex blurred image has beendemodulated.
 5. The method of claim 1, wherein the residual blockscomprise the multiple convolutional calculations, a rectified linearunit activation, and one of the respective skip connections.
 6. Themethod of claim 5, wherein the one of the respective skip connectionsadds a block input to a block output, corresponding to a respectiveresidual block, to provide data continuity between earlier residualblocks and later residual blocks of the CNN.
 7. The method of claim 1,wherein the residual blocks comprise two sequential convolutions of thelayered data set.
 8. The method of claim 1, wherein the CNN comprisesthree consecutive residual blocks of two sequential convolutions.
 9. Themethod of claim 1, wherein the convolutional computations comprise 5×5convolution layers with filter depths of 128 followed by rectifiedlinear unit activations.
 10. The method of claim 1, further comprisingminimizing L1 loss between an output of the CNN and a ground truthimage.
 11. The method of claim 1, further comprising training the CNNwith a ground truth image and a simulated blurred image comprisingsimulated blurring artifacts.
 12. The method of claim 11, furthercomprising forming the simulated image from an in-vivo ground truthimage.
 13. The method of claim 11, further comprising forming thesimulated image with a brain mask based on the ground truth image tosimulate off-resonance changes at an air-tissue interface boundary inthe ground truth image.
 14. The method of claim 13, further comprisingforming the brain mask by applying thresholding techniques to the groundtruth image.
 15. The method of claim 14, further comprising using Otsu'sthresholding technique and/or a convex hull thresholding technique tocreate the brain mask.
 16. The method of claim 15, wherein forming thesimulated image comprises generating a simulated field map by forming acombination of a random 2D polynomial with at least one random 2DGaussian function and applying the combination to the brain mask. 17.The method of claim 16, further comprising generating the simulatedblurred image by applying the simulated field map to the ground truthwith multifrequency interpolation, wherein the multifrequencyinterpolation utilizes a simulated k-space trajectory of the groundtruth image.
 18. The method of claim 16, further comprising utilizingrandom patch cropping as a training augmentation for each simulatedblurred image.
 19. The method of claim 11, further comprising formingthe simulated image with multi-coil data.
 20. The method of claim 19,further comprising forming the simulated image with emulated single coildata based on the multi-coil data.